A proposal to evaluate the management of tuberculosis programs: a qualitative, evaluability assessment in the border region of Brazil and Venezuela

This study aims to analyze the feasibility of building an evaluative model for the management of the Tuberculosis Prevention and Control Program in the State of Roraima, located on the border between Brazil and Venezuela. This is an evaluability assessment, a type of study used as a pre-evaluation of the development and implementation stages of a program, as well as throughout its execution. The study was developed in stages comprising the: (i) definition of the intervention to be analyzed and its objectives and goals; (ii) construction of the intervention logical model; (iii) screening of parties interested in the evaluation; (iv) definition of the evaluative questions; and (v) design of the evaluation matrix. Four priority components were defined for the evaluation: management of the organization and implementation of tuberculosis (TB) prevention and control policy; epidemiological surveillance management; care network management; and management of expected/achieved results. In this model, and based on theoretical references, we defined the necessary resources, activities, outputs, outcomes, and the expected impact for each of the policy management components. The management of the TB control program is feasible for evaluation based on the design of its components, the definition of structure and process indicators, and relevant results for the analysis of the management of TB prevention and control actions, as well as its influence on compliance with the agreed indicators and targets aiming at eradicating the disease by 2035.


Introduction
Tuberculosis (TB) is a disease that persists as a serious global public health problem, responsible for about 10 million people falling ill each year 1 .Despite the existence of several viable protocols for TB control, a number of factors, such as limitations in the quality and effectiveness of programmatic actions developed by health systems for the prevention and control of this disease contribute to its persistence as an important condition, especially in developing countries 2 .
In Brazil, from 2015 to 2021, the total number of TB cases in vulnerable populations, including migrants, increased significantly.From 2015 to 2019, outbreaks of measles, hepatitis A, TB, malaria, syphilis, and leishmaniasis in groups of Venezuelan immigrants living on the border between Brazil and Venezuela, Roraima State 5 , were reported to the Information System for Notifiable Diseases (SINAN, acronym in Portuguese) of the Brazilian Ministry of Health.All the diseases reported in immigrants showed higher numbers than those recorded in the national population living in that territory.
Since 2015, the State of Roraima has been dealing with the intense migration of Venezuelans motivated by the country's political, economic, and social crisis, which began in 2013 6 .Even before the raise in immigration, the State of Roraima already showed significant numbers of TB in its resident population.According to the Epidemiological Bulletin of the Brazilian Ministry of Health, in 2014 Roraima recorded an incidence rate of 29.7 cases/100,000 inhabitants, ranking in the northern Brazilian states with the highest TB incidence 7 .
A total of 2,111 cases of TB were reported in Roraima from 2009 to 2019 in SINAN.Of these, a total of 49 (2.4%) notified cases of TN were individuals from the State of Amazonas, which borders the national territory with the State of Roraima and 18 (10.9%)new cases were reported in immigrants from other countries, most from Venezuela 132 (72.9%) 8 .In 1999, the Brazilian Ministry of Health created the Brazilian National Tuberculosis Control Program (PNCT, acronym in Portuguese), with the objective of expanding TB control actions in the country and reducing the prevalence of the disease in the population 9 .In 2014, the World Health Organization (WHO) approved a new global strategy to fight TB, with the strategic goal of eliminating the disease by 2035 10 .However, achieving these goals depends on a strategic planning of actions based on a situational diagnosis of local needs, with the incorporation of monitoring routines and evaluation of the results achieved 11 .
Even amid the constant implementation of national, state, and municipal health policies and programs, the Brazilian Unified National Health System (SUS, acronym in Portuguese) is facing a global crisis regarding the effective organization and management of health services and actions, increasingly unable to meet the real and specific demands of each health territory, especially in remote regions such as border regions 11,12,13 .
Despite the existence of an international health regulation that guides appropriate surveillance practices at international borders, the guidelines prioritize large-scale events caused by rapidly spreading infectious diseases 14 and this, depending on the priorities of management action, brings the risk of neglecting TB, especially its resistant forms, which should require priority surveillance.
Previous studies have widely proposed evaluating the implementation of health programs, actions, and services 2,6,9 .However, the evaluation of the management of these programs is still incipient and remains a challenge in the field of evaluative research.
Considering the epidemiological profile of TB in the State of Roraima and, more specifically, on the border between Brazil and Venezuela, we identified the need to evaluate the management process of the Tuberculosis Prevention and Control Program in this state, in order to analyze the degree of implementation of policy management and the degree of management influence on the implementation of TB prevention and control actions.
This study aims to examine the feasibility of building an evaluative model for the management of the Tuberculosis Prevention and Control Program in the State of Roraima.

Material and methods
This is an evaluability assessment study.Evaluability assessment is a type of study used as a preevaluation of the development and implementation stages of a program, as well as throughout its execution 15 .The study was developed in five stages: (i) definition of the intervention to be analyzed and its objectives and goals; (ii) construction of the logical model of the intervention; (iii) screening of parties interested in the evaluation; (iv) definition of the evaluative questions; and (v) design of the evaluation matrix.
We identified the Tuberculosis Prevention and Control Program, its objectives and goals as the target of our evaluation, especially regarding management processes.
The logical model allows us to visualize, in a systematic and detailed way, the components of an intervention to be analyzed 16 .By constructing the logical model, it is possible to specifically describe the elements that make up the structure, process, and expected results of the intervention 15 .
To build the logical model of management evaluation, the manuals and technical guidelines recommended by the Brazilian Ministry of Health 10,17,18,19 for implementing TB prevention and control strategies in the territories where they operate were used as theoretical references.In this model, and based on the theoretical references, we defined the necessary resources, activities, outputs, outcomes, and expected impact for each of the policy management components, according to the scheme in Figure 1.
After this phase, the logical model was analyzed by five specialists in health evaluation and planning who independently collaborated to adjust and validate the construct.These specialists included three doctoral professors specialized in health evaluation and two health policy managers interested in the evaluation: one from the health evaluation area and the other from the health surveillance area.Three specialists were contacted via emailing and the other two via face-to-face, individual meetings.At contact, the objectives of the research and the matrix to be validated were explained.The experts only had access to the matrix after agreeing to take part in the research.Anonymity and confidentiality of the raters' identity were preserved.
Four specialists were randomly selected to analyze all the components of the matrix and suggest any necessary adjustments.The acceptability of each item in the matrix components was assessed using a form containing the following answer alternatives: agree, partially agree and suggest adjustments, and disagree.For each item evaluated, a total percentage > 65% agreement between the specialists was considered.In case of a tie between the answers, the 5th judge was responsible for analyzing the questions again and deciding with greater weight on the permanence, adjustment, or exclusion of the item evaluated.The evaluation of the experts was carried out over one month.After the evaluation, the final product was applied in a pilot test in two priority municipalities for managing TB policy in the State of Roraima.
A construct validation of the logical model and matrices was carried out, especially regarding the clarity of the content and the relation of the items to be evaluated with the objectives of the analysis.Once the logical model had been constructed, those interested in the evaluation were listed and the evaluative questions were formulated, especially considering the relevance and feasibility of the information that could be obtained with each question and the relation with the logical model of the intervention to be evaluated.
The matrix of evaluative questions was elaborated based on the theoretical framework used to construct the logical model of the program.Based on the evaluative questions, the evaluation design was built, where the items to be evaluated, the evaluation parameters, and the instruments to be used for each step of the process were categorized (Box 1).Other six matrices already validated in previous studies were also considered and adapted for this study 2,6,20,21,22,23

Results
The theoretical framework enabled the identification of the competences established for the management of the TB control program at the different administrative levels (i.e., state and municipal), as well as the criteria that should be considered for each strategy without implementation.To construct the logical model, we considered the guidelines of pillars 1 (prevention and integrated people-centered care) and 2 (bold policies and support system) of the Brazilian National Plan to End Tuberculosis as a Public Health Problem (2021-2025) and the attributions of each federal entity to comply with the plan 10 .Four priority components were defined for evaluation: management of the organization and implementation of TB prevention and control policy, management of epidemiological surveillance, management of the care network, and management of the expected/obtained results.For each component, strategic activities considered essential for the effective management of the program were added and activities, products, results, and expected impact in terms of TB-related health indicators The following aspects were considered: management profile and qualification, management autonomy and financial support for decision-making, coherence and assertiveness among decisionmaking, program objectives, territory reality, capacity of articulation of the healthcare network, strategic planning culture, and quality of the results obtained in decision-making (Box 3).These analysis items triggered the construction of the judgment matrix and the 34 parameters to be analyzed (Box 1).

Discussion
This pre-evaluation study highlights the relevance of incorporating health evaluation activities into the context of political-institutional management of a health program so that the implementation of health actions can result in the achievement of the proposed goals and objectives, in this particular case, for TB.The management of health services is an administrative practice that aims to optimize the functioning of organizations to obtain results that reflect efficiency in work relationships, effectiveness in achieving objectives and goals, and effectiveness in solving health demands 24 .In the context of the SUS, results-oriented management should adopt evaluation as an activity integrated into public management and the functioning of the political system, using evaluative research as an instrument to support these practices 25 .
Health evaluation should be used as a routine practice for strategic management, aiming at improving the actions and services offered to the population.However, several authors point to the incipient culture of health evaluation in Brazil, especially in the scope of evaluating the management methods implemented for these policies to be carried out 2,24,26 .
The Brazilian Ministry of Health, in its technical guidelines, describes in detail the duties inherent in the management of the TB program, at all federal instances 18 .In this sense, the management of the TB control policy, at all spheres of government, should be based on the theoretical and methodological foundations of health surveillance and the protocols produced for this purpose, and also on a prior evaluation that allows a broad view of the components of the operationalization of actions and identification of unmet needs for improvement and correction in the plan to be developed 27 .
Based on these theoretical references, it was possible to select and describe which activities are essential in health management component so that the intervention can be effectively implemented in the territory.Previous studies related to the evaluation of TB control programs were used as a reference for the construction of the judgment matrix and the evaluation parameters of this study 2,6,20,21,22,23 .Health management components were mainly analyzed and adapted to the evaluative objectives of this research.
The evaluation of actions and public policies in border regions is considered paramount for resolving the specific demands of this context 6 .Considering that TB holds a significant impact on the epidemiological profile of Roraima, which neighbors Venezuela, evaluative items related to the strategic planning of TB control actions at the international borders of the territory were included in the judgment matrix.
The logical model and judgment matrix developed are instruments capable of consolidating the priorities for effective program management and can be considered valid evaluative models for application in various contexts, enabling strategic management and assertive decision-making 21 .In this study, we consider that the priority analysis objectives addressed in each component presented an interdependent relationship capable of allowing the achievement of results and the expected impact of the actions carried out by program management.
Still, we identified some limitations to carry out a validity study.This, however, does not prevent the study from being implemented: the limitation of theoretical references available for Brazil and, more specifically, for Roraima and the dependence on the willingness of managers and other stakeholders to participate in such a study.In addition, because it is a validity study, the validation process was carried out with a more simplistic methodology in order to demonstrate whether the management of the TB program is evaluable.

Final considerations and conclusions
Evaluation studies are fundamental for decision-making on the implementation of an intervention and can be used from the theoretical conception to the measurement of the results obtained.The management of the TB control program is feasible for evaluation based on the design of its components and the definition of structure, process indicators, and relevant results for the analysis of the management of TB prevention and control actions and its influence on the achievement of the agreed indicators and goals: the eradication of the disease by 2035.
This process of evaluating the management of the program in Roraima is relevant when considering the epidemiological profile of TB and the impact of international migration in this context, which demands a specific management approach to the peculiarities inherent to this public health problem in this territory.

Resumen
El objetivo fue analizar la viabilidad de elaborar un modelo de evaluación para la gestión del Programa de Prevención y Control de la Tuberculosis en el Estado de Roraima, que está en la frontera entre Brasil y Venezuela.Se trata de un estudio de evaluabilidad, un modelo de estudio que se utiliza como evaluación previa en la fase de desarrollo e implementación de un programa, así como a lo largo de su ejecución.El desarrollo del estudio se realizó en etapas: (i) definir la intervención a analizar y sus objetivos y metas; (ii) construir el modelo lógico de la intervención; (iii) mapear los interesados en la evaluación; (iv) definir las preguntas de la evaluación; y (v) trazar la matriz de evaluación.Se definieron cuatro componentes prioritarios para la evaluación: la gestión de la organización e implementación de la política de prevención y control de la tuberculosis (TB), la gestión de la vigilancia epidemiológica, la gestión de la red de atención a la salud y la gestión de los resultados esperados/obtenidos.En este modelo, y basándonos en referentes teóricos, definimos los recursos, actividades, productos, resultados y el impacto esperado para cada uno de los componentes de la gestión de políticas.La gestión del programa de control de la TB puede evaluarse a partir del diseño de sus componentes y de la definición de indicadores de estructura y proceso, así como de resultados relevantes y su influencia para el cumplimiento de las metas pactadas, con el fin de erradicar la enfermedad para 2035.

Figure 1
Figure 1Schematic representation of the logical evaluation model of a health policy.
Recognition and use of the main indicators related to TB prevention and control as a planning and monitoring tool, and analyzes them with bulletins and historical series Does not recognize the indicators and does not use them as a management tool Interview Recognizes the indicators but does not use them as a and monitoring of cases of drug-resistant TB in children, pregnant women, and people with comorbidity, among other special situations Does not map and/or monitor cases of areas with higher incidence of the disease in the general population and in special populations, considered to be more vulnerable, including international migrants Does not map areas with higher incidence of TB in the general population in planning and implementing actions in territories and areas with the highest number of new cases reported and under follow-up for clinical TB treatment There is no priority in planning and implementing actions in territories and areas with a higher number of new cases reported and under follow-up for clinical treatment of TB Interview and official documents: work plans, activity reports There is a priority plan for implementing actions in territories and areas with the highest number of new cases reported and under follow-up for clinical treatment of TB 8. Articulates, plans, and/or carries out cross-border TB surveillance actions Elaboration and/or intersectorial and inter-institutional plan of actions and goals that include epidemiological surveillance for TB prevention and control at the international and national borders of the territory There is no action plan for the prevention and control of TB at borders Interview and official documents: work plans, activity reports There is an action plan prepared for the prevention and control of TB at borders, but it is not implemented There is an action plan prepared for the prevention and control of TB at borders, and it is fully implemented (continues) Cad.Saúde Pública 2024; 40(3):e00104823 prevention and control of TB in the territories Prepares and disseminates reports on the activities carried out, goals achieved, and results of indicators established for the prevention and control of TB in the territories supervision to verify the implementation of TB cases notification in all health units at all levels of care in the territories of operation Does not carry out supervision Interview and official documents: work plans, activity reports Carries out punctual supervision or only on request to verify the implementation of TB case notification in all health units at all levels of care, in the territories of operation Periodically carries out supervision to verify the implementation of TB case notification in all health units at all levels of care, in the territories of operation 2. Articulation with the reference laboratory network to provide TB diagnostic tests Preparation and execution of a joint action plan with the laboratory network to ensure the provision of diagnostic imaging tests and other diagnostic tests that are recommended by the PNCT technical regulations (sputum bacilloscopy, culture, X-ray, RMT, sensitivity test, HIV testing, TST, BCG vaccination) There is no action plan and no laboratory network offering laboratory tests for TB diagnosis Interview and official documents: work plans, and implementation of actions to raise awareness among health professionals for active search for respiratory symptoms in the territories and implementation of qualification, updating, and/or professional improvement actions for all health professionals working in the clinical management of TB, at all levels of assistance Does not promote or carry out qualification, updating, and/ or professional improvement actions for all health professionals who work in the clinical management of TB, at all , and control of the distribution of medicines and inputs used in the clinical management of TB with the logistics and distribution centers and pharmacies of health units at all levels of care Planning, supervision, and control of the distribution of medicines used in the clinical management of TB with logistics and distribution centers and pharmacies of health units at all measures for the implementation and supervision of quality control of samples and laboratory analyses carried out in the diagnostic reference laboratories for TB in the territories Does of work plans that include specific health activities, goals, and indicators for special populations or those at greater risk and vulnerability for TB, based on the epidemiological profile of TB in the territories where it operates (including international immigrants), seeking to expand and facilitate access for these groups to the means of prevention, diagnosis, and treatment of TB There is no specific action plan for special populations or populations at greater risk and vulnerability risk of TB (number of new TB cases, divided by population, multiplied by 100,000) of death from TB (number of deaths with underlying cause TB, divided by population, multiplied by 100,000) There was a reduction in the TB mortality rate Data from SINAN and management reports and bulletins There was a maintenance of the TB mortality coefficient There was an increase in the TB mortality of TB treatment and the consequent decrease in disease transmission (total of new cases of pulmonary TB terminated with diagnosis of cure x 100/total of new cases of TB diagnosed) proportion of patients who abandon treatment, remaining absent for more than 60 days after the last visit or remaining without medication for more than 30 days (total of TB cases terminated by treatment abandonment x 100/total of TB cases reported)

5 . 5 .
Integrated planning of actions with the government and other institutions of civil society for actions to be carried out with populations at greater risk of TB, defined.The evaluation components developed are directly related to the structure, process, and results used in health policy evaluation (Box 2).Based on the construction of the logical model, four evaluative questions emerged and were defined to analyze the degree of implementation of the management of the program's structure, process, and results: (1) What actions are taken and what work is carried out by the managing entity to control TB? (2) How are the actions planned and monitored?(3) What is the manager's perception of the impact of migration on the epidemiological profile of TB? (4) What are the challenges faced in managing the TB prevention and control policy?Work plan presenting integrated planning of goals and actions established with other federal, state, and municipal institutions, as well as nongovernmental institutions that work directly with vulnerable populationsINCIDENCE OF TB REDUCTION IN TB MORTALITY PROGRESSIVE ELIMINATION OF TB BY 2035 DOT: directly observed treatment; FHS: Familiy Health Strategy; PNCT: Brazilian National Tuberculosis Control Program; SINAN: Brazilian Information System for Notificable Diseases; TB: tuberculosis.

4 . 3
Planning actions in accordance with the technical-operational guidelines and regulations established for PNCT in Brazil 4. Periodically carries out the quality analysis of the data entered in SINAN 4. Implementation of improvement and qualification actions on TB for professionals who work directly in the diagnosis and treatment of cases (FHS, outpatient clinics, hospitals) 5. Implementation of intersectoral planning of integrated actions with technical areas of related policies for the prevention and control of TB 5. Manages and/or monitors incidence in the territory 6. Integrated planning of actions with the government and other institutions of civil society for actions to be carried out for populations at risk of TB vulnerability, including international migrants 6. Recognizes territories and special populations most vulnerable and at risk of TB, including international migrants 6. Implementation and monitoring of DOT at all levels of care 7. Planning and execution of financial budget for the provision of materials necessary for TB control actions 7. Maps and prioritizes areas with the highest incidence of cases for TB control actions 7. Provides and controls the logistics of medicines and supplies necessary for the clinical management of TB in the territory of operation 4. TB mortality rate in the territory 8. Articulates, plans, and/or carries out cross-border TB surveillance actions 8. Supervision and participation in the action planning of bacteriological diagnosis and quality control at the Dimensions for analyzing the degree of implementation of the management of structure, process and results by component.DOT: directly observed treatment; FHS: Family Health Strategy; PNCT: Brazilian National Tuberculosis Control Program; SINAN: Brazilian Information System for Notificable Diseases; TB: tuberculosis.Cad.Saúde Pública 2024; 40(3):e00104823 Tuberculosis; Migrantes; Salud Fronteriza; Gestión en Salud Submitted on 13/Jun/2023 Final version resubmitted on 14/Dec/2023 Approved on 11/Jan/2024